About this document

Project description

This document is intended both as an open-source record of analyses conducted for one study in our paper entitled “The Effects of Metacognition in Survey Research: Experimental, Cross-Sectional, and Content-Analytic Evidence”, which was published recently in Public Opinion Quarterly, and as an introductory tutorial to Stuctural Topic Modeling (STM). If used for this latter purpose, this document is offered under the same license that governs the dataset used herein (available here). Chiefly, this document is provided “AS IS”, without warranty of any kind, express or implied. That said, if you encounter any issues, please feel free to email the first author using the link at the top of this document.

STMs are described in some detail in the paper. Additional resources are available here and here. Briefly, STMs are a form of unsupervised machine learning used to categorize a corpus of text documents. What differentiates STMs from a “bag of words” or latent Dirichlet allocation approach is the ability to specify document metadata as covariates in the model.

We encourage you to read the paper linked above. A quick synopsis is required to understand the project we undertook. We are interested in whether the language complexity used in public opinion polling differentially affects metacognitive processes, and thus methodologically-pertinent outcomes such as “don’t know” reporting, or abstaining from self-reporting. In study 1, we conducted a survey-experiment among a nationally-representative sample of U.S. adults. Participants were randomly assigned to either an “easy”- or “difficult”-language condition in which we varied the objective complexity of the language used to convey the public opinion questions. In study 2, we collected survey instruments from the Roper Center’s database of public opinion polls (data and analyses coming soon; keep an eye on this website: https://github.com/Matt-Sweitzer/). Using top-line results, we compared question difficulty to rates of “don’t know” reporting. Having found significant results for both studies, we turned our attention to the public opinion polling industry more broadly to see if language complexity varied systematically – suggesting that the methodological problem posed by complex language may be widespread and unconsidered. One of the ways we looked for systematic variance in question complexity is by using a STM to categorize the questions into topics. Given that we had such a large dataset for this latter study, using a computational approach to code topics made this component of our analyses much more feasible.

Session information from the most recent compiling

Before we begin, here is the most up-to-date information about the R session used to generate this file. To run this yourself, you will need to install the R packages stm, stringr, tm, wordcloud, and SnowballC. The latter two packages are only required to plot the wordcloud below and can be skipped for the topic model.

[1] "Wednesday October 16, 2019 - 12:28:26 AM EDT"
R version 3.6.1 (2019-07-05)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Mojave 10.14.6

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] knitr_1.25         wordcloud_2.6      RColorBrewer_1.1-2
[4] tm_0.7-6           NLP_0.2-0          stringr_1.4.0     
[7] stm_1.3.3          SnowballC_0.6.0   

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.2        xml2_1.2.2        magrittr_1.5     
 [4] lattice_0.20-38   highr_0.8         tools_3.6.1      
 [7] parallel_3.6.1    grid_3.6.1        data.table_1.12.2
[10] xfun_0.10         htmltools_0.3.6   yaml_2.2.0       
[13] digest_0.6.21     Matrix_1.2-17     base64enc_0.1-3  
[16] evaluate_0.14     slam_0.1-45       rmarkdown_1.16   
[19] stringi_1.4.3     compiler_3.6.1    jsonlite_1.6     

Analyses

Read in and clean the data

The data for this study is available from my GitHub page.

data<-read.csv(url("https://raw.githubusercontent.com/Matt-Sweitzer/Metacognition_Surveys/master/Study_3/Data/2016PollsFinal.csv"))

Let’s take a look at the head of the data frame:

Date Poll QuestionNum QuestionWord Region N SurveyMethod Likely.vs.All Ease Grade National Geo SurveyNum Topic
2/5/16 Quinnipiac University 1 If the Republican primary for President were being held today, and the candidates were Jeb Bush, Ben Carson, Chris Christie, Ted Cruz, Carly Fiorina, Jim Gilmore, John Kasich, Marco Rubio, and Donald Trump, for whom would you vote? US 1125 Phone Registered 34.69 17.86 1 National 1 6
2/5/16 Quinnipiac University 2 Is your mind made up, or do you think you might change your mind before the primary? US 1125 Phone Registered 90.05 4.92 1 National 1 2
2/5/16 Quinnipiac University 3 Are there any of these candidates you would definitely not support for the Republican nomination for president: Bush, Carson, Christie, Cruz, Fiorina, Gilmore, Kasich, Rubio, or Trump? US 1125 Phone Registered 36.47 11.53 1 National 1 6
2/5/16 Quinnipiac University 4 If the Democratic primary for President were being held today, and the candidates were Hillary Clinton and Bernie Sanders, for whom would you vote? US 1125 Phone Registered 41.48 13.44 1 National 1 6
2/5/16 Quinnipiac University 5 Is your mind made up, or do you think you might change your mind before the primary? US 1125 Phone Registered 90.05 4.92 1 National 1 2
2/5/16 Quinnipiac University 6 If the election for President were being held today, and the candidates were Hillary Clinton the Democrat and Donald Trump the Republican, for whom would you vote? US 1125 Phone Registered 38.43 14.61 1 National 1 6

These variables can be interpreted as follows:

  • Date: character, the date the results were published, or last date of data collection if publication date was not available
  • Poll: character, the polling firm either responsible for data collection or the sponsor of third-party data collection
  • QuestionNum: numeric, indicates the order of questions within the same ballot
  • QuestionWord: character, wording of the question from the survey instrument
    • Note: we manually removed instructions to the surveyor (e.g., pronunciations, response ordering, etc.)
  • Region: character, geographic region of survey sample
  • N: numeric, total sample size
    • Note: some polling firms ask some questions of only some of their respondents (e.g., answered a prior question in a certain way). We opted to encode only the total sample size for every question on the same survey, as subsample sizes were not uniformly reported.
  • SurveyMethod: character (factor), method of data collection – options include “online”, “phone”, and “phone/internet
  • Likely.vs.All: character (factor), class of respondents targeted in sample – options include “All”, “Likely”, and “Registered”.
  • Ease: numeric, Flesch reading ease score of QuestionWord – calculated using the koRpus package in R
  • Grade: numeric, Flesch-Kincaid grade level of QuestionWord – calculated using the koRpus package in R
  • National: numeric (factor), dummy variable collapsing Region – 1 == “US”, else 0
  • Geo: character (factor), alternative collapsing of Region – options include “Community”, “National”, and “State
  • SurveyNum: numeric (factor), indicates which survey instrument each question comes from – should be identical to Date-Poll combination
  • Topic: numeric, stm topic to which the question had the highest fit, \(\theta\) – this is what we will be estimating here!

Now, let’s begin cleaning up the text corpus to make a basic corpus descriptive figure: a wordcloud. First, let’s take the vector and change it to a corpus class:

questionCorp<-Corpus(VectorSource(data$QuestionWord))

Next, lets start cleaning up some of the text. Wordclouds typically don’t have puntucation listed. Also, we might expect that the most popular words in the English language, articles such as “the” or “a”, would also be popular in our corpus – showing them in this figure may not be very informative, so let’s remove those too.

questionCorp<-tm_map(questionCorp, removePunctuation)
questionCorp<-tm_map(questionCorp, removeWords, stopwords('english'))

Finally, some in the natural language processing realm advocate for a process called “stemming”. This takes similar words and removes the suffixes which differentiate them so they can be treated as referring to the same thing. For example, “communicate”, “communication”, and “communicating” would all become “communicat”. This can be somewhat helpful when those slight differentiations create excess noise in the data, or make for an especially sparse term-document matrix. Some of these stemming algorithms are not particularly robust – your mileage may vary. I will not stem them here, but if you would like to stem your own corpus, uncomment the following line of code:

#questionCorp<-tm_map(questionCorp, stemDocument)

Great! Now that the corpus is cleaned, we can take a look at the wordcloud:

  par(mar=c(0,0,0,0))
  wordcloud(questionCorp, max.words=1000, random.order=FALSE)

Clearly, the two lead candidate in the 2016 presidential election, Hillary Clinton and Donald Trump, were both asked about often in surveys leading up to that election. To plot this for yourself (and save the output), use the code below. Additional plotting commands are available, see ?wordcloud for more information.

k-values, search k, and diagnostic plots

Running the STM

Interpreting and exporting results

LS0tCnRpdGxlOiB8Mi0KICBUaGUgRWZmZWN0cyBvZiBNZXRhY29nbml0aW9uIGluIFN1cnZleSBSZXNlYXJjaCAtLSBTdHVkeSAzIChTdHJ1Y3R1cmFsIFRvcGljIE1vZGVsKQphdXRob3I6Ci0gYWZmaWxpYXRpb246IE9oaW8gU3RhdGUgVW5pdmVyc2l0eSwgU2Nob29sIG9mIENvbW11bmljYXRpb24KICBlbWFpbDogbWF0dHN3ZWl0emVyQGdtYWlsLmNvbQogIG5hbWU6IE1hdHRoZXcgRC4gU3dlaXR6ZXIKLSBhZmZpbGlhdGlvbjogT2hpbyBTdGF0ZSBVbml2ZXJzaXR5LCBTY2hvb2wgb2YgQ29tbXVuaWNhdGlvbgogIG5hbWU6IEhpbGxhcnkgQy4gU2h1bG1hbgpkYXRlOiAiYHIgZm9ybWF0KFN5cy50aW1lKCksICclQiAlZSwgJVknKWAiCm91dHB1dDoKICBodG1sX25vdGVib29rOgogICAgY29kZV9mb2xkaW5nOiBub25lCiMgICAgc2VsZl9jb250YWluZWQ6IEZBTFNFCiMgICAgdGhlbWU6IHlldGkKIyAgICB0b2M6IHllcwojICAgIHRvY19kZXB0aDogMwotLS0KCiMgQWJvdXQgdGhpcyBkb2N1bWVudAoKIyMgUHJvamVjdCBkZXNjcmlwdGlvbgoKVGhpcyBkb2N1bWVudCBpcyBpbnRlbmRlZCBib3RoIGFzIGFuIG9wZW4tc291cmNlIHJlY29yZCBvZiBhbmFseXNlcyBjb25kdWN0ZWQgZm9yIG9uZSBzdHVkeSBpbiBvdXIgcGFwZXIgZW50aXRsZWQgWyJUaGUgRWZmZWN0cyBvZiBNZXRhY29nbml0aW9uIGluIFN1cnZleSBSZXNlYXJjaDogRXhwZXJpbWVudGFsLCBDcm9zcy1TZWN0aW9uYWwsIGFuZCBDb250ZW50LUFuYWx5dGljIEV2aWRlbmNlIl0oaHR0cHM6Ly9naXRodWIuY29tL01hdHQtU3dlaXR6ZXIvUGFwZXJzL2Jsb2IvbWFzdGVyL1N3ZWl0emVyU2h1bG1hbjIwMTgucGRmKSwgd2hpY2ggd2FzIHB1Ymxpc2hlZCByZWNlbnRseSBpbiBQdWJsaWMgT3BpbmlvbiBRdWFydGVybHksIGFuZCBhcyBhbiBpbnRyb2R1Y3RvcnkgdHV0b3JpYWwgdG8gU3R1Y3R1cmFsIFRvcGljIE1vZGVsaW5nIChTVE0pLiBJZiB1c2VkIGZvciB0aGlzIGxhdHRlciBwdXJwb3NlLCB0aGlzIGRvY3VtZW50IGlzIG9mZmVyZWQgdW5kZXIgdGhlIHNhbWUgbGljZW5zZSB0aGF0IGdvdmVybnMgdGhlIGRhdGFzZXQgdXNlZCBoZXJlaW4gKGF2YWlsYWJsZSBbaGVyZV0oaHR0cHM6Ly9naXRodWIuY29tL01hdHQtU3dlaXR6ZXIvTWV0YWNvZ25pdGlvbl9TdXJ2ZXlzL2Jsb2IvbWFzdGVyL0xJQ0VOU0UpKS4gQ2hpZWZseSwgdGhpcyBkb2N1bWVudCBpcyBwcm92aWRlZCAiQVMgSVMiLCB3aXRob3V0IHdhcnJhbnR5IG9mIGFueSBraW5kLCBleHByZXNzIG9yIGltcGxpZWQuIFRoYXQgc2FpZCwgaWYgeW91IGVuY291bnRlciBhbnkgaXNzdWVzLCBwbGVhc2UgZmVlbCBmcmVlIHRvIGVtYWlsIHRoZSBmaXJzdCBhdXRob3IgdXNpbmcgdGhlIGxpbmsgYXQgdGhlIHRvcCBvZiB0aGlzIGRvY3VtZW50LgoKU1RNcyBhcmUgZGVzY3JpYmVkIGluIHNvbWUgZGV0YWlsIGluIHRoZSBwYXBlci4gQWRkaXRpb25hbCByZXNvdXJjZXMgYXJlIGF2YWlsYWJsZSBbaGVyZV0oaHR0cDovL3d3dy5zdHJ1Y3R1cmFsdG9waWNtb2RlbC5jb20vKSBhbmQgW2hlcmVdKGh0dHA6Ly9zY2hvbGFyLmhhcnZhcmQuZWR1L2ZpbGVzL2R0aW5nbGV5L2ZpbGVzL3RvcGljbW9kZWxzb3BlbmVuZGVkZXhwZXJpbWVudHMucGRmKS4gQnJpZWZseSwgU1RNcyBhcmUgYSBmb3JtIG9mIHVuc3VwZXJ2aXNlZCBtYWNoaW5lIGxlYXJuaW5nIHVzZWQgdG8gY2F0ZWdvcml6ZSBhIGNvcnB1cyBvZiB0ZXh0IGRvY3VtZW50cy4gV2hhdCBkaWZmZXJlbnRpYXRlcyBTVE1zIGZyb20gYSAiYmFnIG9mIHdvcmRzIiBvciBsYXRlbnQgRGlyaWNobGV0IGFsbG9jYXRpb24gYXBwcm9hY2ggaXMgdGhlIGFiaWxpdHkgdG8gc3BlY2lmeSBkb2N1bWVudCBtZXRhZGF0YSBhcyBjb3ZhcmlhdGVzIGluIHRoZSBtb2RlbC4KCldlIGVuY291cmFnZSB5b3UgdG8gcmVhZCB0aGUgcGFwZXIgbGlua2VkIGFib3ZlLiBBIHF1aWNrIHN5bm9wc2lzIGlzIHJlcXVpcmVkIHRvIHVuZGVyc3RhbmQgdGhlIHByb2plY3Qgd2UgdW5kZXJ0b29rLiBXZSBhcmUgaW50ZXJlc3RlZCBpbiB3aGV0aGVyIHRoZSBsYW5ndWFnZSBjb21wbGV4aXR5IHVzZWQgaW4gcHVibGljIG9waW5pb24gcG9sbGluZyBkaWZmZXJlbnRpYWxseSBhZmZlY3RzIG1ldGFjb2duaXRpdmUgcHJvY2Vzc2VzLCBhbmQgdGh1cyBtZXRob2RvbG9naWNhbGx5LXBlcnRpbmVudCBvdXRjb21lcyBzdWNoIGFzICJkb24ndCBrbm93IiByZXBvcnRpbmcsIG9yIGFic3RhaW5pbmcgZnJvbSBzZWxmLXJlcG9ydGluZy4gSW4gc3R1ZHkgMSwgd2UgY29uZHVjdGVkIGEgc3VydmV5LWV4cGVyaW1lbnQgYW1vbmcgYSBuYXRpb25hbGx5LXJlcHJlc2VudGF0aXZlIHNhbXBsZSBvZiBVLlMuIGFkdWx0cy4gUGFydGljaXBhbnRzIHdlcmUgcmFuZG9tbHkgYXNzaWduZWQgdG8gZWl0aGVyIGFuICJlYXN5Ii0gb3IgImRpZmZpY3VsdCItbGFuZ3VhZ2UgY29uZGl0aW9uIGluIHdoaWNoIHdlIHZhcmllZCB0aGUgb2JqZWN0aXZlIGNvbXBsZXhpdHkgb2YgdGhlIGxhbmd1YWdlIHVzZWQgdG8gY29udmV5IHRoZSBwdWJsaWMgb3BpbmlvbiBxdWVzdGlvbnMuIEluIHN0dWR5IDIsIHdlIGNvbGxlY3RlZCBzdXJ2ZXkgaW5zdHJ1bWVudHMgZnJvbSB0aGUgUm9wZXIgQ2VudGVyJ3MgZGF0YWJhc2Ugb2YgcHVibGljIG9waW5pb24gcG9sbHMgKGRhdGEgYW5kIGFuYWx5c2VzIGNvbWluZyBzb29uOyBrZWVwIGFuIGV5ZSBvbiB0aGlzIHdlYnNpdGU6IFtodHRwczovL2dpdGh1Yi5jb20vTWF0dC1Td2VpdHplci9dKGh0dHBzOi8vZ2l0aHViLmNvbS9NYXR0LVN3ZWl0emVyLykpLiBVc2luZyB0b3AtbGluZSByZXN1bHRzLCB3ZSBjb21wYXJlZCBxdWVzdGlvbiBkaWZmaWN1bHR5IHRvIHJhdGVzIG9mICJkb24ndCBrbm93IiByZXBvcnRpbmcuIEhhdmluZyBmb3VuZCBzaWduaWZpY2FudCByZXN1bHRzIGZvciBib3RoIHN0dWRpZXMsIHdlIHR1cm5lZCBvdXIgYXR0ZW50aW9uIHRvIHRoZSBwdWJsaWMgb3BpbmlvbiBwb2xsaW5nIGluZHVzdHJ5IG1vcmUgYnJvYWRseSB0byBzZWUgaWYgbGFuZ3VhZ2UgY29tcGxleGl0eSB2YXJpZWQgc3lzdGVtYXRpY2FsbHkgLS0gc3VnZ2VzdGluZyB0aGF0IHRoZSBtZXRob2RvbG9naWNhbCBwcm9ibGVtIHBvc2VkIGJ5IGNvbXBsZXggbGFuZ3VhZ2UgbWF5IGJlIHdpZGVzcHJlYWQgYW5kIHVuY29uc2lkZXJlZC4gT25lIG9mIHRoZSB3YXlzIHdlIGxvb2tlZCBmb3Igc3lzdGVtYXRpYyB2YXJpYW5jZSBpbiBxdWVzdGlvbiBjb21wbGV4aXR5IGlzIGJ5IHVzaW5nIGEgU1RNIHRvIGNhdGVnb3JpemUgdGhlIHF1ZXN0aW9ucyBpbnRvIHRvcGljcy4gR2l2ZW4gdGhhdCB3ZSBoYWQgc3VjaCBhIGxhcmdlIGRhdGFzZXQgZm9yIHRoaXMgbGF0dGVyIHN0dWR5LCB1c2luZyBhIGNvbXB1dGF0aW9uYWwgYXBwcm9hY2ggdG8gY29kZSB0b3BpY3MgbWFkZSB0aGlzIGNvbXBvbmVudCBvZiBvdXIgYW5hbHlzZXMgbXVjaCBtb3JlIGZlYXNpYmxlLgoKIyMgU2Vzc2lvbiBpbmZvcm1hdGlvbiBmcm9tIHRoZSBtb3N0IHJlY2VudCBjb21waWxpbmcKCkJlZm9yZSB3ZSBiZWdpbiwgaGVyZSBpcyB0aGUgbW9zdCB1cC10by1kYXRlIGluZm9ybWF0aW9uIGFib3V0IHRoZSBSIHNlc3Npb24gdXNlZCB0byBnZW5lcmF0ZSB0aGlzIGZpbGUuIFRvIHJ1biB0aGlzIHlvdXJzZWxmLCB5b3Ugd2lsbCBuZWVkIHRvIGluc3RhbGwgdGhlIFIgcGFja2FnZXMgYHN0bWAsIGBzdHJpbmdyYCwgYHRtYCwgYHdvcmRjbG91ZGAsIGFuZCBgU25vd2JhbGxDYC4gVGhlIGxhdHRlciB0d28gcGFja2FnZXMgYXJlIG9ubHkgcmVxdWlyZWQgdG8gcGxvdCB0aGUgd29yZGNsb3VkIGJlbG93IGFuZCBjYW4gYmUgc2tpcHBlZCBmb3IgdGhlIHRvcGljIG1vZGVsLgoKYGBge3IgZWNobz1GLCBtZXNzYWdlPUZ9CmxpYnJhcnkoU25vd2JhbGxDKQpsaWJyYXJ5KHN0bSkKbGlicmFyeShzdHJpbmdyKQpsaWJyYXJ5KHRtKQpsaWJyYXJ5KHdvcmRjbG91ZCkKYGBgCgpgYGB7ciBlY2hvPUZ9CmZvcm1hdChTeXMudGltZSgpLCAiJUEgJUIgJWUsICVZIC0gJUk6JU06JVMgJXAgJVoiKQpzZXNzaW9uSW5mbygpCmBgYAoKIyBBbmFseXNlcwoKIyMgUmVhZCBpbiBhbmQgY2xlYW4gdGhlIGRhdGEKClRoZSBkYXRhIGZvciB0aGlzIHN0dWR5IGlzIGF2YWlsYWJsZSBmcm9tIG15IEdpdEh1YiBwYWdlLgoKYGBge3J9CmRhdGE8LXJlYWQuY3N2KHVybCgiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL01hdHQtU3dlaXR6ZXIvTWV0YWNvZ25pdGlvbl9TdXJ2ZXlzL21hc3Rlci9TdHVkeV8zL0RhdGEvMjAxNlBvbGxzRmluYWwuY3N2IikpCmBgYAoKTGV0J3MgdGFrZSBhIGxvb2sgYXQgdGhlIGhlYWQgb2YgdGhlIGRhdGEgZnJhbWU6CgpgYGB7ciBlY2hvPUYsIHJlc3VsdHM9J2FzaXMnfQpsaWJyYXJ5KGtuaXRyKQprYWJsZShoZWFkKGRhdGEpKQpgYGAKClRoZXNlIHZhcmlhYmxlcyBjYW4gYmUgaW50ZXJwcmV0ZWQgYXMgZm9sbG93czoKCiogYERhdGVgOiBjaGFyYWN0ZXIsIHRoZSBkYXRlIHRoZSByZXN1bHRzIHdlcmUgcHVibGlzaGVkLCBvciBsYXN0IGRhdGUgb2YgZGF0YSBjb2xsZWN0aW9uIGlmIHB1YmxpY2F0aW9uIGRhdGUgd2FzIG5vdCBhdmFpbGFibGUKKiBgUG9sbGA6IGNoYXJhY3RlciwgdGhlIHBvbGxpbmcgZmlybSBlaXRoZXIgcmVzcG9uc2libGUgZm9yIGRhdGEgY29sbGVjdGlvbiBvciB0aGUgc3BvbnNvciBvZiB0aGlyZC1wYXJ0eSBkYXRhIGNvbGxlY3Rpb24KKiBgUXVlc3Rpb25OdW1gOiBudW1lcmljLCBpbmRpY2F0ZXMgdGhlIG9yZGVyIG9mIHF1ZXN0aW9ucyAqd2l0aGluKiB0aGUgc2FtZSBiYWxsb3QKKiBgUXVlc3Rpb25Xb3JkYDogY2hhcmFjdGVyLCB3b3JkaW5nIG9mIHRoZSBxdWVzdGlvbiBmcm9tIHRoZSBzdXJ2ZXkgaW5zdHJ1bWVudAogICsgKk5vdGUqOiB3ZSBtYW51YWxseSByZW1vdmVkIGluc3RydWN0aW9ucyB0byB0aGUgc3VydmV5b3IgKGUuZy4sIHByb251bmNpYXRpb25zLCByZXNwb25zZSBvcmRlcmluZywgZXRjLikKKiBgUmVnaW9uYDogY2hhcmFjdGVyLCBnZW9ncmFwaGljIHJlZ2lvbiBvZiBzdXJ2ZXkgc2FtcGxlCiogYE5gOiBudW1lcmljLCB0b3RhbCBzYW1wbGUgc2l6ZQogICsgKk5vdGUqOiBzb21lIHBvbGxpbmcgZmlybXMgYXNrIHNvbWUgcXVlc3Rpb25zIG9mIG9ubHkgc29tZSBvZiB0aGVpciByZXNwb25kZW50cyAoZS5nLiwgYW5zd2VyZWQgYSBwcmlvciBxdWVzdGlvbiBpbiBhIGNlcnRhaW4gd2F5KS4gV2Ugb3B0ZWQgdG8gZW5jb2RlIG9ubHkgdGhlICp0b3RhbCogc2FtcGxlIHNpemUgZm9yIGV2ZXJ5IHF1ZXN0aW9uIG9uIHRoZSBzYW1lIHN1cnZleSwgYXMgc3Vic2FtcGxlIHNpemVzIHdlcmUgbm90IHVuaWZvcm1seSByZXBvcnRlZC4KKiBgU3VydmV5TWV0aG9kYDogY2hhcmFjdGVyIChmYWN0b3IpLCBtZXRob2Qgb2YgZGF0YSBjb2xsZWN0aW9uIC0tIG9wdGlvbnMgaW5jbHVkZSAiYG9ubGluZWAiLCAiYHBob25lYCIsIGFuZCAiYHBob25lL2ludGVybmV0YCIKKiBgTGlrZWx5LnZzLkFsbGA6IGNoYXJhY3RlciAoZmFjdG9yKSwgY2xhc3Mgb2YgcmVzcG9uZGVudHMgdGFyZ2V0ZWQgaW4gc2FtcGxlIC0tIG9wdGlvbnMgaW5jbHVkZSAiYEFsbGAiLCAiYExpa2VseWAiLCBhbmQgImBSZWdpc3RlcmVkYCIuCiogYEVhc2VgOiBudW1lcmljLCBGbGVzY2ggcmVhZGluZyBlYXNlIHNjb3JlIG9mIGBRdWVzdGlvbldvcmRgIC0tIGNhbGN1bGF0ZWQgdXNpbmcgdGhlIGBrb1JwdXNgIHBhY2thZ2UgaW4gUgoqIGBHcmFkZWA6IG51bWVyaWMsIEZsZXNjaC1LaW5jYWlkIGdyYWRlIGxldmVsIG9mIGBRdWVzdGlvbldvcmRgIC0tIGNhbGN1bGF0ZWQgdXNpbmcgdGhlIGBrb1JwdXNgIHBhY2thZ2UgaW4gUgoqIGBOYXRpb25hbGA6IG51bWVyaWMgKGZhY3RvciksIGR1bW15IHZhcmlhYmxlIGNvbGxhcHNpbmcgYFJlZ2lvbmAgLS0gMSA9PSAiVVMiLCBlbHNlIDAKKiBgR2VvYDogY2hhcmFjdGVyIChmYWN0b3IpLCBhbHRlcm5hdGl2ZSBjb2xsYXBzaW5nIG9mIGBSZWdpb25gIC0tIG9wdGlvbnMgaW5jbHVkZSAiYENvbW11bml0eWAiLCAiYE5hdGlvbmFsYCIsIGFuZCAiYFN0YXRlYCIKKiBgU3VydmV5TnVtYDogbnVtZXJpYyAoZmFjdG9yKSwgaW5kaWNhdGVzIHdoaWNoIHN1cnZleSBpbnN0cnVtZW50IGVhY2ggcXVlc3Rpb24gY29tZXMgZnJvbSAtLSBzaG91bGQgYmUgaWRlbnRpY2FsIHRvIGBEYXRlYC1gUG9sbGAgY29tYmluYXRpb24KKiBgVG9waWNgOiBudW1lcmljLCBzdG0gdG9waWMgdG8gd2hpY2ggdGhlIHF1ZXN0aW9uIGhhZCB0aGUgaGlnaGVzdCBmaXQsICRcdGhldGEkIC0tIHRoaXMgaXMgd2hhdCB3ZSB3aWxsIGJlIGVzdGltYXRpbmcgaGVyZSEKCk5vdywgbGV0J3MgYmVnaW4gY2xlYW5pbmcgdXAgdGhlIHRleHQgY29ycHVzIHRvIG1ha2UgYSBiYXNpYyBjb3JwdXMgZGVzY3JpcHRpdmUgZmlndXJlOiBhIHdvcmRjbG91ZC4gRmlyc3QsIGxldCdzIHRha2UgdGhlIHZlY3RvciBhbmQgY2hhbmdlIGl0IHRvIGEgY29ycHVzIGNsYXNzOgoKYGBge3J9CnF1ZXN0aW9uQ29ycDwtQ29ycHVzKFZlY3RvclNvdXJjZShkYXRhJFF1ZXN0aW9uV29yZCkpCmBgYAoKTmV4dCwgbGV0cyBzdGFydCBjbGVhbmluZyB1cCBzb21lIG9mIHRoZSB0ZXh0LiBXb3JkY2xvdWRzIHR5cGljYWxseSBkb24ndCBoYXZlIHB1bnR1Y2F0aW9uIGxpc3RlZC4gQWxzbywgd2UgbWlnaHQgZXhwZWN0IHRoYXQgdGhlIG1vc3QgcG9wdWxhciB3b3JkcyBpbiB0aGUgRW5nbGlzaCBsYW5ndWFnZSwgYXJ0aWNsZXMgc3VjaCBhcyAidGhlIiBvciAiYSIsIHdvdWxkIGFsc28gYmUgcG9wdWxhciBpbiBvdXIgY29ycHVzIC0tIHNob3dpbmcgdGhlbSBpbiB0aGlzIGZpZ3VyZSBtYXkgbm90IGJlIHZlcnkgaW5mb3JtYXRpdmUsIHNvIGxldCdzIHJlbW92ZSB0aG9zZSB0b28uCgpgYGB7ciB3YXJuaW5nPUZ9CnF1ZXN0aW9uQ29ycDwtdG1fbWFwKHF1ZXN0aW9uQ29ycCwgcmVtb3ZlUHVuY3R1YXRpb24pCnF1ZXN0aW9uQ29ycDwtdG1fbWFwKHF1ZXN0aW9uQ29ycCwgcmVtb3ZlV29yZHMsIHN0b3B3b3JkcygnZW5nbGlzaCcpKQpgYGAKCkZpbmFsbHksIHNvbWUgaW4gdGhlIG5hdHVyYWwgbGFuZ3VhZ2UgcHJvY2Vzc2luZyByZWFsbSBhZHZvY2F0ZSBmb3IgYSBwcm9jZXNzIGNhbGxlZCAic3RlbW1pbmciLiBUaGlzIHRha2VzIHNpbWlsYXIgd29yZHMgYW5kIHJlbW92ZXMgdGhlIHN1ZmZpeGVzIHdoaWNoIGRpZmZlcmVudGlhdGUgdGhlbSBzbyB0aGV5IGNhbiBiZSB0cmVhdGVkIGFzIHJlZmVycmluZyB0byB0aGUgc2FtZSB0aGluZy4gRm9yIGV4YW1wbGUsICJjb21tdW5pY2F0ZSIsICJjb21tdW5pY2F0aW9uIiwgYW5kICJjb21tdW5pY2F0aW5nIiB3b3VsZCBhbGwgYmVjb21lICJjb21tdW5pY2F0Ii4gVGhpcyBjYW4gYmUgc29tZXdoYXQgaGVscGZ1bCB3aGVuIHRob3NlIHNsaWdodCBkaWZmZXJlbnRpYXRpb25zIGNyZWF0ZSBleGNlc3Mgbm9pc2UgaW4gdGhlIGRhdGEsIG9yIG1ha2UgZm9yIGFuIGVzcGVjaWFsbHkgc3BhcnNlIHRlcm0tZG9jdW1lbnQgbWF0cml4LiBTb21lIG9mIHRoZXNlIHN0ZW1taW5nIGFsZ29yaXRobXMgYXJlIG5vdCBwYXJ0aWN1bGFybHkgcm9idXN0IC0tIHlvdXIgbWlsZWFnZSBtYXkgdmFyeS4gSSB3aWxsIG5vdCBzdGVtIHRoZW0gaGVyZSwgYnV0IGlmIHlvdSB3b3VsZCBsaWtlIHRvIHN0ZW0geW91ciBvd24gY29ycHVzLCB1bmNvbW1lbnQgdGhlIGZvbGxvd2luZyBsaW5lIG9mIGNvZGU6CgpgYGB7cn0KI3F1ZXN0aW9uQ29ycDwtdG1fbWFwKHF1ZXN0aW9uQ29ycCwgc3RlbURvY3VtZW50KQpgYGAKCkdyZWF0ISBOb3cgdGhhdCB0aGUgY29ycHVzIGlzIGNsZWFuZWQsIHdlIGNhbiB0YWtlIGEgbG9vayBhdCB0aGUgd29yZGNsb3VkOgoKYGBge3IgZXZhbD1GLCBmaWcuYWxpZ249J2NlbnRlcicsIGZpZy53aWR0aD04LCBmaWcuaGVpZ2h0PTh9CiAgcGFyKG1hcj1jKDAsMCwwLDApKQogIHdvcmRjbG91ZChxdWVzdGlvbkNvcnAsIG1heC53b3Jkcz0xMDAwLCByYW5kb20ub3JkZXI9RkFMU0UpCmBgYAohW10oaHR0cHM6Ly9naXRodWIuY29tL01hdHQtU3dlaXR6ZXIvTWV0YWNvZ25pdGlvbl9TdXJ2ZXlzL3Jhdy9tYXN0ZXIvU3R1ZHlfMy9BbmFseXNlcy93b3JkY2xvdWQucG5nKQoKQ2xlYXJseSwgdGhlIHR3byBsZWFkIGNhbmRpZGF0ZSBpbiB0aGUgMjAxNiBwcmVzaWRlbnRpYWwgZWxlY3Rpb24sIEhpbGxhcnkgQ2xpbnRvbiBhbmQgRG9uYWxkIFRydW1wLCB3ZXJlIGJvdGggYXNrZWQgYWJvdXQgb2Z0ZW4gaW4gc3VydmV5cyBsZWFkaW5nIHVwIHRvIHRoYXQgZWxlY3Rpb24uIFRvIHBsb3QgdGhpcyBmb3IgeW91cnNlbGYgKGFuZCBzYXZlIHRoZSBvdXRwdXQpLCB1c2UgdGhlIGNvZGUgYmVsb3cuIEFkZGl0aW9uYWwgcGxvdHRpbmcgY29tbWFuZHMgYXJlIGF2YWlsYWJsZSwgc2VlIGA/d29yZGNsb3VkYCBmb3IgbW9yZSBpbmZvcm1hdGlvbi4KCiMjICprKi12YWx1ZXMsIHNlYXJjaCAqayosIGFuZCBkaWFnbm9zdGljIHBsb3RzCgojIyBSdW5uaW5nIHRoZSBTVE0KCiMjIEludGVycHJldGluZyBhbmQgZXhwb3J0aW5nIHJlc3VsdHM=